> **Part 14 of 20** in the DSPX-2604 re-cut. Base branch:
`dspx-2604-13-unresolved-kas`.
>
> This stack replaces #3782 / #3865 / #3921, which stay open and
untouched
> until it lands. Nothing here is a rebase of those branches — the work
was
> re-cut from the ticket so each PR stands on its own.

### Proposed Changes

`sdk/experimental/tdf` carried its own copies of the manifest and
assertion
types -- `Manifest`, `Segment`, `KeyAccess`, `Assertion`, `Statement`,
`AssertionKey` and the rest -- structurally identical to the ones in
`sdk`
but distinct to the type system, so anything crossing the boundary
needed
conversion. Two copies of the JWT signing and verification logic also
had
to be kept in step by hand.

Replaces both files' definitions with type aliases. `sdk` owns the
definitions; this package re-exports them. Every exported name and every
method survives: `Assertion.Sign` / `Verify` / `GetHash`,
`Statement.UnmarshalJSON`, `AssertionKey.IsEmpty` / `Algorithm`,
`AssertionVerificationKeys.Get` / `IsEmpty` and the five `String()`
methods
all come along with the aliased types, so importers compile unchanged. A
manifest produced here can now be handed to the stable SDK without
conversion, which is what the follow-up delegation needs.

Two deliberate non-aliases:

`Policy`, `PolicyBody` and `PolicyAttribute` stay local.
`sdk.PolicyObject`
declares `Body` as an anonymous struct over an unexported element type,
so
there is no nameable sdk equivalent to alias to. Exporting those in
`sdk`
first would make the alias possible; that is a separate change.

`IntegrityAlgorithm` stays a distinct `int` type.
`sdk.IntegrityAlgorithm`
is itself `= int`, so no method can be attached to it, and aliasing
would
silently drop `String()` from this package's public API. The underlying
values match, so the two convert freely.

`kSplitKeyType`, `kPolicyBindingAlg`, `kGMACPayloadLength` and
`calculateSignature` are retained verbatim: this package still builds
its
own manifests and they have callers in `writer.go` and `key_access.go`.
The change that removes those callers removes these too.

No behavior change. The package's existing tests pass unmodified.

### Checklist

- [ ] I have added or updated unit tests
- [ ] I have added or updated integration tests (if appropriate)
- [ ] I have added or updated documentation

### Testing Instructions

```
cd sdk && go test ./experimental/... -race
```

No behavior change — the point is that the package's existing tests pass
unmodified against aliased types.

<details>
<summary><b>The full DSPX-2604 stack — 20 PRs</b></summary>

| # | PR | Based on |
|---|----|----------|
| 01 | #3930 chore: bump go.work toolchain to go1.25.12 and simplify an
rt_test condition | `main` |
| 02 | #3931 feat(sdk): make the zipstream clock injectable for
deterministic ZIP output | `main` |
| 03 | #3932 fix(sdk): reject a zipstream write set that omits segment 0
| #3931 |
| 04 | #3933 fix(sdk): map ReadAt plaintext offsets from cumulative
segment sizes | `main` |
| 05 | #3934 chore(sdk): extract integrityAlgorithmString,
createPolicyBinding, signAssertions | `main` |
| 06 | #3935 chore(sdk): add direct tests for createKeyAccess,
encryptMetadata and tdfSalt | `main` |
| 07 | #3936 fix(sdk): fill each segment with io.ReadFull and size the
buffer to the input | `main` |
| 08 | #3937 chore(cli): move streaming IO helpers into pkg | `main` |
| 09 | #3938 fix(cli): stream encrypt instead of buffering the whole
payload | #3937 |
| 10 | #3939 fix(cli): stream decrypt and inspect instead of buffering |
#3938 |
| 11 | #3940 feat(sdk): add a chunked segment writer (experimental) |
`dspx-2604-base-11` = #3932 + #3934 + #3935 |
| 12 | #3941 fix(sdk): stop GetManifest from splitting the key under the
lock | #3940 |
| 13 | #3942 fix(sdk): reject a chunked split naming a KAS with no
resolved public key | #3941 |
| 14 | #3943 chore(sdk): alias experimental/tdf manifest and assertion
types | #3942 |
| 15 | #3944 fix(sdk): emit spec-compliant key access in
experimental/tdf and delegate Writer | #3943 |
| 16 | #3945 feat(sdk): accept io.Reader in CreateTDF and drop the 64 GB
payload cap | #3936 |
| 17 | #3946 chore(sdk): rewrite CreateTDF on top of the chunked writer
| `dspx-2604-base-17` = #3944 + #3945 |
| 18 | #3947 chore(sdk): drop dead TDFConfig fields and deprecate the
TDFFormat enum | #3946 |
| 19 | #3948 fix(cli): drop the encrypt-side stdin spool |
`dspx-2604-base-19` = #3947 + #3939 |
| 20 | #3949 feat(sdk): graduate the chunked writer to stable API |
#3948 |

**Reviewable in parallel right now**, since they sit directly on `main`
and depend on
nothing else: 01, 02, 04, 05, 06, 07, 08.

**Why three PRs have a `dspx-2604-base-*` base.** A GitHub PR takes one
base branch,
but 11, 17 and 19 each build on more than one parent. The `base-*`
branches are empty
merge commits that exist only to join those parents so the PR diff shows
exactly its
own change and nothing else. They contain no code, have no PR of their
own, and go
away once their parents land — retarget the child onto `main` at that
point.

**Wants a cross-SDK xtest run before merge:** 15, 17 (and therefore 20).
They touch
the KAS wire format.

**Red checks you may see are network flakes, not this stack.** Four
distinct ones hit
this batch and all clear on re-run: `golangci-lint config verify` timing
out on
`https://golangci-lint.run/.../golangci.v2.8.jsonschema.json` (fails the
whole `go
(<module>)` job and fail-fast cancels its siblings), the bats installer
getting a 403,
Docker Hub timing out on `keycloak/keycloak:26.4`, and `buf` reporting
"the server
hosted at that remote is unavailable" while the Java SDK generates
sources. The
`govulncheck` step also emits `##[error]` annotations against the
go1.25.11 stdlib, but
it is `continue-on-error: true` and never fails a job — 01 bumps the
toolchain and
clears those annotations.

</details>


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Compatibility**
* Experimental TDF manifest and assertion types now align with the
stable SDK, enabling direct interoperability without type conversion.
* Existing assertion and manifest behavior is provided through the
stable SDK definitions.

* **Documentation**
* Clarified the relationship between the experimental TDF package and
the stable SDK.
* Updated documentation for missing assertion verification keys to
reflect current behavior.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Signed-off-by: Dave Mihalcik <dmihalcik@virtru.com>

> **Part 19 of 20** in the DSPX-2604 re-cut. Base branch:
`dspx-2604-base-19`.
>
> This stack replaces #3782 / #3865 / #3921, which stay open and
untouched
> until it lands. Nothing here is a rebase of those branches — the work
was
> re-cut from the ticket so each PR stands on its own.

### Proposed Changes

DSPX-4499 fixed encrypt's OOM by streaming, but CreateTDF still required
an
io.ReadSeeker, so piped stdin had to be spooled to a temporary file
first. That
traded the whole-payload allocation for a disk write and a writable
TMPDIR --
better, but not the point. Now that CreateTDF takes an io.Reader, the
pipe goes
straight to the SDK.

A file is still opened seekably, on purpose. The SDK measures a seekable
payload and keeps the archive in the compact ZIP32 layout; an
unmeasurable one
has to be ZIP64, because the choice is baked into the payload's local
file
header before the first segment goes out. So `encrypt file.txt` is
unchanged
byte for byte, and `... | encrypt` produces a slightly larger TDF than
it did
when it was spooled. That is the trade, and it is the right way round:
nobody
should need a writable temp directory to encrypt a stream.

MIME sniffing is what made this more than a deletion. It reads the first
megabyte and previously seeked back to zero, which a pipe cannot do.
Wrapping
the input in a bufio.Reader is not an option either -- that hides the
Seeker
and would silently flip every file encrypt to ZIP64. detectMimeType now
returns
a reader alongside the type: the same reader for a seekable input,
rewound; the
sniffed prefix pushed back with io.MultiReader for anything else. A
megabyte in
memory at most, and only when --mime-type was not given.

Testing: TestDetectMimeTypePreservesThePayload now runs each case twice,
once
over a reader whose Seek method is hidden, and asserts the payload
arrives
whole either way. TestDetectMimeTypeKeepsSeekability guards the ZIP32
layout
directly, since nothing else would fail if a file came back wrapped. e2e
gains
"encrypt measures a file and streams a pipe", which reads the local file
header's extra field length to tell the two layouts apart -- both forms
round-trip, so a quiet return to spooling would show up nowhere else.

### Checklist

- [x] I have added or updated unit tests
- [x] I have added or updated integration tests (if appropriate)
- [x] I have added or updated documentation

### Testing Instructions

```
cd otdfctl && go test ./... -race
```

e2e, against a running platform:

```
cd otdfctl && bats e2e/streaming.bats
```

The file is tagged `payload_streaming` and runs in its own pass ahead of
the
parallel batch — see #3939 for why that ordering is load-bearing (the
platform
base key `key-base.bats` sets cannot be un-set, and breaks every
unattributed
encrypt scheduled after it).

The case to look at is `encrypt measures a file and streams a pipe`,
which
asserts the ZIP layout rather than the round trip: a file must stay
ZIP32
(extra field length 0 at offset 28 of the payload's local file header)
and a
pipe must be ZIP64 (non-zero). A quiet return to spooling would show up
nowhere
else, since both forms round-trip fine.

<details>
<summary><b>The full DSPX-2604 stack — 20 PRs</b></summary>

| # | PR | Based on |
|---|----|----------|
| 01 | #3930 chore: bump go.work toolchain to go1.25.12 and simplify an
rt_test condition | `main` |
| 02 | #3931 feat(sdk): make the zipstream clock injectable for
deterministic ZIP output | `main` |
| 03 | #3932 fix(sdk): reject a zipstream write set that omits segment 0
| #3931 |
| 04 | #3933 fix(sdk): map ReadAt plaintext offsets from cumulative
segment sizes | `main` |
| 05 | #3934 chore(sdk): extract integrityAlgorithmString,
createPolicyBinding, signAssertions | `main` |
| 06 | #3935 chore(sdk): add direct tests for createKeyAccess,
encryptMetadata and tdfSalt | `main` |
| 07 | #3936 fix(sdk): fill each segment with io.ReadFull and size the
buffer to the input | `main` |
| 08 | #3937 chore(cli): move streaming IO helpers into pkg | `main` |
| 09 | #3938 fix(cli): stream encrypt instead of buffering the whole
payload | #3937 |
| 10 | #3939 fix(cli): stream decrypt and inspect instead of buffering |
#3938 |
| 11 | #3940 feat(sdk): add a chunked segment writer (experimental) |
`dspx-2604-base-11` = #3932 + #3934 + #3935 |
| 12 | #3941 fix(sdk): stop GetManifest from splitting the key under the
lock | #3940 |
| 13 | #3942 fix(sdk): reject a chunked split naming a KAS with no
resolved public key | #3941 |
| 14 | #3943 chore(sdk): alias experimental/tdf manifest and assertion
types | #3942 |
| 15 | #3944 fix(sdk): emit spec-compliant key access in
experimental/tdf and delegate Writer | #3943 |
| 16 | #3945 feat(sdk): accept io.Reader in CreateTDF and drop the 64 GB
payload cap | #3936 |
| 17 | #3946 chore(sdk): rewrite CreateTDF on top of the chunked writer
| `dspx-2604-base-17` = #3944 + #3945 |
| 18 | #3947 chore(sdk): drop dead TDFConfig fields and deprecate the
TDFFormat enum | #3946 |
| 19 | #3948 fix(cli): drop the encrypt-side stdin spool |
`dspx-2604-base-19` = #3947 + #3939 |
| 20 | #3949 feat(sdk): graduate the chunked writer to stable API |
#3948 |

**Reviewable in parallel right now**, since they sit directly on `main`
and depend on
nothing else: 01, 02, 04, 05, 06, 07, 08.

**Why three PRs have a `dspx-2604-base-*` base.** A GitHub PR takes one
base branch,
but 11, 17 and 19 each build on more than one parent. The `base-*`
branches are empty
merge commits that exist only to join those parents so the PR diff shows
exactly its
own change and nothing else. They contain no code, have no PR of their
own, and go
away once their parents land — retarget the child onto `main` at that
point.

**Wants a cross-SDK xtest run before merge:** 15, 17 (and therefore 20).
They touch
the KAS wire format.

**Red checks you may see are network flakes, not this stack.** Four
distinct ones hit
this batch and all clear on re-run: `golangci-lint config verify` timing
out on
`https://golangci-lint.run/.../golangci.v2.8.jsonschema.json` (fails the
whole `go
(<module>)` job and fail-fast cancels its siblings), the bats installer
getting a 403,
Docker Hub timing out on `keycloak/keycloak:26.4`, and `buf` reporting
"the server
hosted at that remote is unavailable" while the Java SDK generates
sources. The
`govulncheck` step also emits `##[error]` annotations against the
go1.25.11 stdlib, but
it is `continue-on-error: true` and never fails a job — 01 bumps the
toolchain and
clears those annotations.

</details>

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Encryption now supports piped and other non-seekable input, including
larger streams.
* Inputs that cannot be measured in advance produce ZIP64 output;
seekable files continue to use ZIP32 when applicable.
* Regular-file redirects can be read directly without being copied to a
temporary file.
  * Decryption now supports TDFs supplied through pipes and redirects.
* Encrypted output remains compatible with the existing decryption flow.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

Signed-off-by: Dave Mihalcik <dmihalcik@virtru.com>
(cherry picked from commit 5603c2e)

…ap (#3945)

> **Part 16 of 20** in the DSPX-2604 re-cut. Base branch:
`dspx-2604-07-readfull`.
>
> This stack replaces #3782 / #3865 / #3921, which stay open and
untouched
> until it lands. Nothing here is a rebase of those branches — the work
was
> re-cut from the ticket so each PR stands on its own.

### Proposed Changes

CreateTDF and CreateTDFContext took an io.ReadSeeker, so a caller with a
pipe,
a socket, or any other one-pass source had to spool the whole payload to
disk
or memory first. That is the block DSPX-2604 exists to remove: the
Everfox
re-wrap pipeline hands us a stream it cannot rewind. Both now take an
io.Reader and consume it from its current position through EOF.

Seekability was only ever used to measure the input. The length still
matters,
but it is now resolved rather than required:

- WithInputSize(n) declares it outright, for a reader that cannot report
it;
- failing that, a reader that happens to implement io.Seeker is probed,
and
    the cursor restored to wherever it was;
  - failing both, the payload is unmeasurable and is read until it ends.

The one thing an unmeasurable payload gives up is the compact ZIP32
layout.
The ZIP64 decision is baked into the payload's local file header, which
is
emitted ahead of the first segment, so it cannot be revisited once the
archive
has started; a payload that might exceed a 32-bit offset has to be
written as
ZIP64 from the outset. WithInputSize exists to buy that back — declaring
the
length of a piped payload keeps it in ZIP32 when it fits.

The read loop no longer computes a segment count up front. It reads a
buffer
at a time until EOF, which is what makes an unknown length workable, and
happens to be the same code path for a short final segment. An empty
payload
still produces one empty segment. The segment count is still passed to
the
archive writer when it is known, because that is what keeps a large
declared
count from being clamped to a one-segment capacity hint.

Three behavior changes worth calling out. The first is the only one that
can
silently shorten a payload; the other two fail loudly.

- A seekable reader is no longer rewound. The old code seeked to the end
to
measure and then back to byte 0, so it encrypted the whole file no
matter
where the caller had left the cursor. resolveInputSize saves the current
position and restores that one, so the payload is whatever remains from
    there. Concretely: a caller that sniffs the first 512 bytes for
content-type detection and then hands the same *os.File to CreateTDF
used
to get the whole file and now gets the file minus 512 bytes — no compile
error, no runtime error, just a shorter TDF that looks complete. Reading
from the current position is the right semantics for an io.Reader-shaped
API, and `a seekable reader is encrypted from its current position` pins
    it, but a caller relying on the rewind has to seek to 0 itself now.

- The 64 GB cap (maxFileSizeSupported/errFileTooLarge) is gone. It could
only ever be enforced on a measurable payload, so keeping it would have
meant `encrypt bigfile` failing where `encrypt < bigfile` succeeded.
Both
    were unexported; nothing outside the package referenced them.

- A declared size is exact, not an upper bound. A reader that reaches
EOF
    early now fails the call with errInputShorterThanDeclared instead of
returning a TDF that is silently short of the payload the caller asked
to
encrypt. Reading still stops at the declared size if the reader has
more.

What bounds a payload now, with the cap gone. maxPayloadSegments refuses
a size
needing more than MaxInt32 segments, but segmentCount is the only place
it is
enforced and that runs only when the length resolves — an unmeasurable
stream
has no segment ceiling. Underneath that, the real limit is manifest
memory: one
manifest segment and one archive-writer entry per segment, all live
until
finalize, roughly 500k entries per terabyte at the 2 MiB default.
Neither is
reachable today — 4 PiB at that segment size — so this is a note about
where
the wall moved to, not a regression. DSPX-4905 tracks bounding and
measuring
both.

Testing: Test_CreateTDF_StreamingInput covers the three measurement
modes
across empty, sub-segment, exact-multiple, and partial-final-segment
payloads,
asserting the ZIP64 choice, the segment count, and a full round trip
through
LoadTDF. Test_CreateTDF_InputSizeBounds covers the negative, over-long,
short,
and mid-stream-start cases. Both guards were mutation-checked: removing
the
io.LimitReader fails "declared size bounds the read", and dropping the
unknown-size ZIP64 rule fails every unmeasurable case.

### Checklist

- [x] I have added or updated unit tests
- [ ] I have added or updated integration tests (if appropriate)
- [ ] I have added or updated documentation

### Testing Instructions

```
cd sdk && go test ./... -race
```

`Test_CreateTDF_StreamingInput` and `Test_CreateTDF_InputSizeBounds` are
the
new coverage. Both guards were mutation-checked: removing the
`io.LimitReader`
fails "declared size bounds the read", and dropping the unknown-size
ZIP64 rule
fails every unmeasurable case.

<details>
<summary><b>The full DSPX-2604 stack — 20 PRs</b></summary>

| # | PR | Based on |
|---|----|----------|
| 01 | #3930 chore: bump go.work toolchain to go1.25.12 and simplify an
rt_test condition | `main` |
| 02 | #3931 feat(sdk): make the zipstream clock injectable for
deterministic ZIP output | `main` |
| 03 | #3932 fix(sdk): reject a zipstream write set that omits segment 0
| #3931 |
| 04 | #3933 fix(sdk): map ReadAt plaintext offsets from cumulative
segment sizes | `main` |
| 05 | #3934 chore(sdk): extract integrityAlgorithmString,
createPolicyBinding, signAssertions | `main` |
| 06 | #3935 chore(sdk): add direct tests for createKeyAccess,
encryptMetadata and tdfSalt | `main` |
| 07 | #3936 fix(sdk): fill each segment with io.ReadFull and size the
buffer to the input | `main` |
| 08 | #3937 chore(cli): move streaming IO helpers into pkg | `main` |
| 09 | #3938 fix(cli): stream encrypt instead of buffering the whole
payload | #3937 |
| 10 | #3939 fix(cli): stream decrypt and inspect instead of buffering |
#3938 |
| 11 | #3940 feat(sdk): add a chunked segment writer (experimental) |
`dspx-2604-base-11` = #3932 + #3934 + #3935 |
| 12 | #3941 fix(sdk): stop GetManifest from splitting the key under the
lock | #3940 |
| 13 | #3942 fix(sdk): reject a chunked split naming a KAS with no
resolved public key | #3941 |
| 14 | #3943 chore(sdk): alias experimental/tdf manifest and assertion
types | #3942 |
| 15 | #3944 fix(sdk): emit spec-compliant key access in
experimental/tdf and delegate Writer | #3943 |
| 16 | #3945 feat(sdk): accept io.Reader in CreateTDF and drop the 64 GB
payload cap | #3936 |
| 17 | #3946 chore(sdk): rewrite CreateTDF on top of the chunked writer
| `dspx-2604-base-17` = #3944 + #3945 |
| 18 | #3947 chore(sdk): drop dead TDFConfig fields and deprecate the
TDFFormat enum | #3946 |
| 19 | #3948 fix(cli): drop the encrypt-side stdin spool |
`dspx-2604-base-19` = #3947 + #3939 |
| 20 | #3949 feat(sdk): graduate the chunked writer to stable API |
#3948 |

**Reviewable in parallel right now**, since they sit directly on `main`
and depend on
nothing else: 01, 02, 04, 05, 06, 07, 08.

**Why three PRs have a `dspx-2604-base-*` base.** A GitHub PR takes one
base branch,
but 11, 17 and 19 each build on more than one parent. The `base-*`
branches are empty
merge commits that exist only to join those parents so the PR diff shows
exactly its
own change and nothing else. They contain no code, have no PR of their
own, and go
away once their parents land — retarget the child onto `main` at that
point.

**Wants a cross-SDK xtest run before merge:** 15, 17 (and therefore 20).
They touch
the KAS wire format.

**Red checks you may see are network flakes, not this stack.** Four
distinct ones hit
this batch and all clear on re-run: `golangci-lint config verify` timing
out on
`https://golangci-lint.run/.../golangci.v2.8.jsonschema.json` (fails the
whole `go
(<module>)` job and fail-fast cancels its siblings), the bats installer
getting a 403,
Docker Hub timing out on `keycloak/keycloak:26.4`, and `buf` reporting
"the server
hosted at that remote is unavailable" while the Java SDK generates
sources. The
`govulncheck` step also emits `##[error]` annotations against the
go1.25.11 stdlib, but
it is `continue-on-error: true` and never fails a job — 01 bumps the
toolchain and
clears those annotations.

</details>

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* TDF creation now supports streaming inputs, including non-seekable
readers such as pipes and network streams.
* Added `WithInputSize` to declare the exact payload size when it cannot
be determined automatically.
* Payloads are processed in segments, supporting unknown sizes and ZIP64
archives when needed.

* **Bug Fixes**
* Invalid or inaccurate declared sizes now result in clear errors,
including when input ends too early.
  * Payloads exceeding the supported segment limit are rejected.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

Signed-off-by: Dave Mihalcik <dmihalcik@virtru.com>
(cherry picked from commit 20ff505)

> **Part 19 of 20** in the DSPX-2604 re-cut. Base branch:
`dspx-2604-base-19`.
>
> This stack replaces #3782 / #3865 / #3921, which stay open and
untouched
> until it lands. Nothing here is a rebase of those branches — the work
was
> re-cut from the ticket so each PR stands on its own.

### Proposed Changes

DSPX-4499 fixed encrypt's OOM by streaming, but CreateTDF still required
an
io.ReadSeeker, so piped stdin had to be spooled to a temporary file
first. That
traded the whole-payload allocation for a disk write and a writable
TMPDIR --
better, but not the point. Now that CreateTDF takes an io.Reader, the
pipe goes
straight to the SDK.

A file is still opened seekably, on purpose. The SDK measures a seekable
payload and keeps the archive in the compact ZIP32 layout; an
unmeasurable one
has to be ZIP64, because the choice is baked into the payload's local
file
header before the first segment goes out. So `encrypt file.txt` is
unchanged
byte for byte, and `... | encrypt` produces a slightly larger TDF than
it did
when it was spooled. That is the trade, and it is the right way round:
nobody
should need a writable temp directory to encrypt a stream.

MIME sniffing is what made this more than a deletion. It reads the first
megabyte and previously seeked back to zero, which a pipe cannot do.
Wrapping
the input in a bufio.Reader is not an option either -- that hides the
Seeker
and would silently flip every file encrypt to ZIP64. detectMimeType now
returns
a reader alongside the type: the same reader for a seekable input,
rewound; the
sniffed prefix pushed back with io.MultiReader for anything else. A
megabyte in
memory at most, and only when --mime-type was not given.

Testing: TestDetectMimeTypePreservesThePayload now runs each case twice,
once
over a reader whose Seek method is hidden, and asserts the payload
arrives
whole either way. TestDetectMimeTypeKeepsSeekability guards the ZIP32
layout
directly, since nothing else would fail if a file came back wrapped. e2e
gains
"encrypt measures a file and streams a pipe", which reads the local file
header's extra field length to tell the two layouts apart -- both forms
round-trip, so a quiet return to spooling would show up nowhere else.

### Checklist

- [x] I have added or updated unit tests
- [x] I have added or updated integration tests (if appropriate)
- [x] I have added or updated documentation

### Testing Instructions

```
cd otdfctl && go test ./... -race
```

e2e, against a running platform:

```
cd otdfctl && bats e2e/streaming.bats
```

The file is tagged `payload_streaming` and runs in its own pass ahead of
the
parallel batch — see #3939 for why that ordering is load-bearing (the
platform
base key `key-base.bats` sets cannot be un-set, and breaks every
unattributed
encrypt scheduled after it).

The case to look at is `encrypt measures a file and streams a pipe`,
which
asserts the ZIP layout rather than the round trip: a file must stay
ZIP32
(extra field length 0 at offset 28 of the payload's local file header)
and a
pipe must be ZIP64 (non-zero). A quiet return to spooling would show up
nowhere
else, since both forms round-trip fine.

<details>
<summary><b>The full DSPX-2604 stack — 20 PRs</b></summary>

| # | PR | Based on |
|---|----|----------|
| 01 | #3930 chore: bump go.work toolchain to go1.25.12 and simplify an
rt_test condition | `main` |
| 02 | #3931 feat(sdk): make the zipstream clock injectable for
deterministic ZIP output | `main` |
| 03 | #3932 fix(sdk): reject a zipstream write set that omits segment 0
| #3931 |
| 04 | #3933 fix(sdk): map ReadAt plaintext offsets from cumulative
segment sizes | `main` |
| 05 | #3934 chore(sdk): extract integrityAlgorithmString,
createPolicyBinding, signAssertions | `main` |
| 06 | #3935 chore(sdk): add direct tests for createKeyAccess,
encryptMetadata and tdfSalt | `main` |
| 07 | #3936 fix(sdk): fill each segment with io.ReadFull and size the
buffer to the input | `main` |
| 08 | #3937 chore(cli): move streaming IO helpers into pkg | `main` |
| 09 | #3938 fix(cli): stream encrypt instead of buffering the whole
payload | #3937 |
| 10 | #3939 fix(cli): stream decrypt and inspect instead of buffering |
#3938 |
| 11 | #3940 feat(sdk): add a chunked segment writer (experimental) |
`dspx-2604-base-11` = #3932 + #3934 + #3935 |
| 12 | #3941 fix(sdk): stop GetManifest from splitting the key under the
lock | #3940 |
| 13 | #3942 fix(sdk): reject a chunked split naming a KAS with no
resolved public key | #3941 |
| 14 | #3943 chore(sdk): alias experimental/tdf manifest and assertion
types | #3942 |
| 15 | #3944 fix(sdk): emit spec-compliant key access in
experimental/tdf and delegate Writer | #3943 |
| 16 | #3945 feat(sdk): accept io.Reader in CreateTDF and drop the 64 GB
payload cap | #3936 |
| 17 | #3946 chore(sdk): rewrite CreateTDF on top of the chunked writer
| `dspx-2604-base-17` = #3944 + #3945 |
| 18 | #3947 chore(sdk): drop dead TDFConfig fields and deprecate the
TDFFormat enum | #3946 |
| 19 | #3948 fix(cli): drop the encrypt-side stdin spool |
`dspx-2604-base-19` = #3947 + #3939 |
| 20 | #3949 feat(sdk): graduate the chunked writer to stable API |
#3948 |

**Reviewable in parallel right now**, since they sit directly on `main`
and depend on
nothing else: 01, 02, 04, 05, 06, 07, 08.

**Why three PRs have a `dspx-2604-base-*` base.** A GitHub PR takes one
base branch,
but 11, 17 and 19 each build on more than one parent. The `base-*`
branches are empty
merge commits that exist only to join those parents so the PR diff shows
exactly its
own change and nothing else. They contain no code, have no PR of their
own, and go
away once their parents land — retarget the child onto `main` at that
point.

**Wants a cross-SDK xtest run before merge:** 15, 17 (and therefore 20).
They touch
the KAS wire format.

**Red checks you may see are network flakes, not this stack.** Four
distinct ones hit
this batch and all clear on re-run: `golangci-lint config verify` timing
out on
`https://golangci-lint.run/.../golangci.v2.8.jsonschema.json` (fails the
whole `go
(<module>)` job and fail-fast cancels its siblings), the bats installer
getting a 403,
Docker Hub timing out on `keycloak/keycloak:26.4`, and `buf` reporting
"the server
hosted at that remote is unavailable" while the Java SDK generates
sources. The
`govulncheck` step also emits `##[error]` annotations against the
go1.25.11 stdlib, but
it is `continue-on-error: true` and never fails a job — 01 bumps the
toolchain and
clears those annotations.

</details>

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Encryption now supports piped and other non-seekable input, including
larger streams.
* Inputs that cannot be measured in advance produce ZIP64 output;
seekable files continue to use ZIP32 when applicable.
* Regular-file redirects can be read directly without being copied to a
temporary file.
  * Decryption now supports TDFs supplied through pipes and redirects.
* Encrypted output remains compatible with the existing decryption flow.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

Signed-off-by: Dave Mihalcik <dmihalcik@virtru.com>
(cherry picked from commit 5603c2e)

> **Part 10 of 20** in the DSPX-2604 re-cut. Base branch:
`dspx-2604-09-stream-encrypt`.
>
> This stack replaces opentdf#3782 / opentdf#3865 / opentdf#3921, which stay open and
untouched
> until it lands. Nothing here is a rebase of those branches — the work
was
> re-cut from the ticket so each PR stands on its own.

### Proposed Changes

`otdfctl decrypt` read the whole TDF into memory, handed the slice to
DecryptBytes, which accumulated the whole plaintext in a bytes.Buffer,
and then
-- for stdout -- called Buffer.String(), allocating a third full copy.
Peak RSS
was roughly 3.6x the payload; a 1 GiB file cost ~3.7 GiB of RAM and a
large
enough file simply OOMed on a machine with plenty of disk for it.

The plaintext now streams from the SDK reader to the destination.
Handler.Decrypt
takes an io.ReadSeeker and an io.Writer, with DecryptOptions replacing
the
positional parameter list, and inspect reaches the manifest through the
same
seekable reader rather than buffering the archive to get at its tail.

io.Copy is what does the streaming, and it does so only because
sdk.Reader
implements WriteTo, which decrypts one segment at a time. Its Read
delegates to
ReadAt, which grows an internal bytes.Buffer holding every segment
decrypted so
far -- so dropping WriteTo would silently restore the old memory profile
with no
test failure to show for it. A compile-time assertion pins the
interface.

Removes MaxFileSize. The 10 GB cap existed to bound RAM; the real limit
is the
SDK maxFileSizeSupported at 64 GiB, which enforces itself.

Output to a file is atomic, as on the encrypt side: the plaintext goes
to a
temporary sibling and is renamed into place only on success. Since
cli.ExitWithError calls os.Exit and skips deferred functions, the
spooled input
and the partial output are discarded explicitly on every exit path --
including
inspect's success path, which exits through ExitWithJSON.

e2e coverage lands in a new otdfctl/e2e/streaming.bats rather than in
encrypt-decrypt.bats, which carries a file-level skip pending the
namespaced-subject-mappings migration and would have swallowed the new
cases
without running them. Nothing in the new file needs an entitlement, so
it needs
no policy fixtures: the round-trips use no attributes, and the two
failure cases
are forced with an unresolvable attribute FQN and a KAS allowlist that
excludes
the platform. As of this change it is the only e2e coverage of encrypt,
decrypt
and inspect that actually executes in CI.

The file is tagged payload_streaming and action.yaml gives it its own
pass
ahead of the parallel batch. That ordering is load-bearing, not
tidiness. An
encrypt with no attributes falls back to the platform base key, and
key-base.bats sets one pointing at https://test-kas-for-base-keys.com,
which
does not resolve. It cannot put things back afterwards: a base key can
be
replaced but never cleared, so every unattributed encrypt scheduled
after that
file yields a TDF nothing can decrypt. Under --jobs 4 the file order is
nondeterministic, so overlapping the two made this suite flaky rather
than
merely broken -- which is how it presented, a different subset of
round-trips
failing per run. Running alone also keeps the 1 GiB peak-RSS case from
measuring itself against three neighbours competing for the same memory.

That leak is worth closing on its own -- encrypt-decrypt.bats walks into
it the
day its skip is lifted -- but the fix belongs with the file that opens
it
rather than here.

### Checklist

- [x] I have added or updated unit tests
- [x] I have added or updated integration tests (if appropriate)
- [x] I have added or updated documentation

### Testing Instructions

```
cd otdfctl && go test ./... -race
```

e2e, against a running platform:

```
cd otdfctl && bats --tap e2e --filter-tags payload_streaming
```

The first CI run of this file failed 308–311 and 317, all of them the
cases
that need a successful decrypt. Cause was not the code under test: an
encrypt
with no attributes falls back to the platform base key, and
`key-base.bats`
sets one pointing at `https://test-kas-for-base-keys.com`, which does
not
resolve — and cannot unset it, because a base key can only be replaced.
Under
`--jobs 4` the file order is nondeterministic, so which subset failed
varied
per run. Fixed here by tagging the file `payload_streaming` and giving
it its
own pass before the parallel batch. Tag arithmetic checks out: 14 + 10 +
330 =
354, the same total as before.

The memory case needs GNU `time` (`gtime` on macOS) and skips without
it. It
allocates a 1 GiB file; peak RSS was ~3.6 GiB per command before this
change
and the assertion threshold is 512 MiB.

<details>
<summary><b>The full DSPX-2604 stack — 20 PRs</b></summary>

| # | PR | Based on |
|---|----|----------|
| 01 | opentdf#3930 chore: bump go.work toolchain to go1.25.12 and simplify an
rt_test condition | `main` |
| 02 | opentdf#3931 feat(sdk): make the zipstream clock injectable for
deterministic ZIP output | `main` |
| 03 | opentdf#3932 fix(sdk): reject a zipstream write set that omits segment 0
| opentdf#3931 |
| 04 | opentdf#3933 fix(sdk): map ReadAt plaintext offsets from cumulative
segment sizes | `main` |
| 05 | opentdf#3934 chore(sdk): extract integrityAlgorithmString,
createPolicyBinding, signAssertions | `main` |
| 06 | opentdf#3935 chore(sdk): add direct tests for createKeyAccess,
encryptMetadata and tdfSalt | `main` |
| 07 | opentdf#3936 fix(sdk): fill each segment with io.ReadFull and size the
buffer to the input | `main` |
| 08 | opentdf#3937 chore(cli): move streaming IO helpers into pkg | `main` |
| 09 | opentdf#3938 fix(cli): stream encrypt instead of buffering the whole
payload | opentdf#3937 |
| 10 | opentdf#3939 fix(cli): stream decrypt and inspect instead of buffering |
opentdf#3938 |
| 11 | opentdf#3940 feat(sdk): add a chunked segment writer (experimental) |
`dspx-2604-base-11` = opentdf#3932 + opentdf#3934 + opentdf#3935 |
| 12 | opentdf#3941 fix(sdk): stop GetManifest from splitting the key under the
lock | opentdf#3940 |
| 13 | opentdf#3942 fix(sdk): reject a chunked split naming a KAS with no
resolved public key | opentdf#3941 |
| 14 | opentdf#3943 chore(sdk): alias experimental/tdf manifest and assertion
types | opentdf#3942 |
| 15 | opentdf#3944 fix(sdk): emit spec-compliant key access in
experimental/tdf and delegate Writer | opentdf#3943 |
| 16 | opentdf#3945 feat(sdk): accept io.Reader in CreateTDF and drop the 64 GB
payload cap | opentdf#3936 |
| 17 | opentdf#3946 chore(sdk): rewrite CreateTDF on top of the chunked writer
| `dspx-2604-base-17` = opentdf#3944 + opentdf#3945 |
| 18 | opentdf#3947 chore(sdk): drop dead TDFConfig fields and deprecate the
TDFFormat enum | opentdf#3946 |
| 19 | opentdf#3948 fix(cli): drop the encrypt-side stdin spool |
`dspx-2604-base-19` = opentdf#3947 + opentdf#3939 |
| 20 | opentdf#3949 feat(sdk): graduate the chunked writer to stable API |
opentdf#3948 |

**Reviewable in parallel right now**, since they sit directly on `main`
and depend on
nothing else: 01, 02, 04, 05, 06, 07, 08.

**Why three PRs have a `dspx-2604-base-*` base.** A GitHub PR takes one
base branch,
but 11, 17 and 19 each build on more than one parent. The `base-*`
branches are empty
merge commits that exist only to join those parents so the PR diff shows
exactly its
own change and nothing else. They contain no code, have no PR of their
own, and go
away once their parents land — retarget the child onto `main` at that
point.

**Wants a cross-SDK xtest run before merge:** 15, 17 (and therefore 20).
They touch
the KAS wire format.

**Red checks you may see are network flakes, not this stack.** Four
distinct ones hit
this batch and all clear on re-run: `golangci-lint config verify` timing
out on
`https://golangci-lint.run/.../golangci.v2.8.jsonschema.json` (fails the
whole `go
(<module>)` job and fail-fast cancels its siblings), the bats installer
getting a 403,
Docker Hub timing out on `keycloak/keycloak:26.4`, and `buf` reporting
"the server
hosted at that remote is unavailable" while the Java SDK generates
sources. The
`govulncheck` step also emits `##[error]` annotations against the
go1.25.11 stdlib, but
it is `continue-on-error: true` and never fails a job — 01 bumps the
toolchain and
clears those annotations.

</details>


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

- **New Features**
- Decryption now streams files and standard input, reducing memory usage
for large payloads.
- Decrypted output supports standard output, device paths, and
symbolic-link destinations.
- Failed operations clean up temporary data and avoid leaving incomplete
output files.

- **Bug Fixes**
- Improved handling and validation of empty or invalid encrypted input.
  - Existing destination files are preserved when decryption fails.
- Expanded coverage for streaming, cleanup, special destinations, and
large-payload memory usage.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Signed-off-by: Dave Mihalcik <dmihalcik@virtru.com>

…4087)

Jira: https://virtru.atlassian.net/browse/DSPX-4590

The read-side counterpart to opentdf#3936, which fixed the same class of bug on
the
write side. Independent of every open stack — no PR in DSPX-2604 (opentdf#3941,
opentdf#3943-opentdf#3949) touches `sdk/internal/zipstream`, and opentdf#4043 leaves
`readBytes`
alone — so this sits directly on `main` and can land in any order.

## The bug

`readBytes` issued a single `Read` and, on the nil-error path, returned
the
whole `make()`'d buffer regardless of how many bytes actually arrived:

```go
buf := make([]byte, size)
n, err := readerSeeker.Read(buf)
if errors.Is(err, io.EOF) { return buf[:n], io.EOF }
if err != nil { return buf[:n], ... }
return buf, nil   // full length, zero-padded, n discarded
```

`io.Reader` explicitly permits `0 < n < len(p)` with a nil error, so the
tail
of the buffer comes back as zeros that never existed in the archive —
reported as success.

Demonstrated against a valid fixture with a one-byte-at-a-time
`ReadSeeker`:

```
want "payload bytes"
got  "p\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00"  err=<nil>  len=13
```

`bytes.Reader` never reads short, which is why no existing test caught
it.
But `NewReader` takes a caller-supplied `io.ReadSeeker`, and the
implementations that *do* read short in practice — HTTP range requests,
network filesystems — are exactly the ones large archives get served
from.

The `ErrSegSizeMismatch` guard in `tdf.go` (`sdk/tdf.go:1114`) cannot
catch
this either: `make()` already gave the buffer the length that check is
looking for, so the check is comparing a value that is correct by
construction. A silently zero-padded segment reaches AES-GCM and fails
there instead, several layers from the cause.

## Why this is the same fix as opentdf#3936

opentdf#3936 ("fill each segment with io.ReadFull and size the buffer to the
input")
made this argument for `CreateTDFContext`'s encrypt loop:

> io.Reader.Read is permitted to return fewer bytes than the caller
asked for
> without erroring… A `*bytes.Reader` or `*os.File` on a local disk
rarely
> returns short, which is why this has held up, but any wrapping
ReadSeeker —
> a decompressor, a network-backed store, an instrumented reader — can
> trigger it and there is nothing wrong with the input when it does.

Same defect, opposite consequence. On the write side a short read made
the
loop *fail loudly* on valid input (`io.ReadSeeker.Read size mismatch`).
Here
it *silently corrupts*: no error, full-length buffer, invented bytes.
The
noisy direction got fixed; the quiet one was left, and it is the worse
of the
two. This closes it.

## The fix

`io.ReadFull`, which retries the short read and reports
`io.ErrUnexpectedEOF` when the archive really is truncated. Two related
changes follow from it:

- **No data is returned alongside an error.** The old code handed back a
  partial buffer with a bare `io.EOF` — a trap for a caller doing the
reflexive `errors.Is(err, io.EOF)` → "normal end of stream", which would
  silently accept a truncated payload. All four production callers
(`sdk.go:489`, `tdf.go:864`, `tdf.go:987`, `tdf.go:1109`) already treat
the
  error as fatal, so nothing depends on the old shape.
- **`io.EOF` is normalized to `io.ErrUnexpectedEOF`.** `io.ReadFull`
reports
the bare sentinel when it reads *nothing* — a read starting exactly at
the
end of a truncated archive — and `io.ErrUnexpectedEOF` only when it got
part of the request. Both mean the same thing to a caller that asked for
a
byte count taken from the central directory, and letting the first one
out
  would have reopened the trap above. Zero-length reads are unaffected;
  `io.ReadFull` returns nil for them even at EOF. (Thanks @​opencode for
  catching this.)
- **A zero-length read at the end of the archive now succeeds.** Any
`Read`
on an exhausted `bytes.Reader` reports `io.EOF`, including a zero-length
  one, so an empty entry stored last used to fail to read back.

Error *classification* is deliberately left alone — `readBytes` still
does
not wrap `errZipFormat`. Eight other sites in this file have the same
issue
and that belongs in one follow-up, not here.

## Reading this next to opentdf#3945

opentdf#3945 goes the other way on the same sentinel: it makes the encrypt loop
*tolerate* `io.EOF` and `io.ErrUnexpectedEOF` from `io.ReadFull`, while
this
PR makes `io.ErrUnexpectedEOF` fatal. Both are right, because the
contracts
differ. The write side has no declared length — a short final read just
means
the caller's stream ended. The read side has a length from the central
directory, so a short read means the archive is truncated.

The zero-length change above is also the mirror of opentdf#3945's empty-payload
handling ("An empty payload still gets one empty segment").
`zeroLenOKReader`
in `tdf_segment_defaults_test.go` exists because `SDK.CreateTDF` cannot
currently encrypt a genuinely empty `io.Reader`; opentdf#3945 fixes the write
half
and this fixes the read half. Worth rechecking whether that workaround
can be
deleted once both have landed — not assuming it, since the comment there
describes more than one quirk.

## Tests

`chunkedSeeker` in `reader_test.go` serves at most N bytes per `Read`
and
reports no error for the short ones, standing in for the range-backed
reader
`bytes.Reader` cannot model. Five cases: short reads reassembled, a
partially
satisfiable read rejected with `io.ErrUnexpectedEOF` and no partial
buffer, a
read starting at EOF rejected with the same sentinel rather than
`io.EOF`, a
zero-length read at EOF succeeding, and the same short read driven
end-to-end
through `ReadAllFileData`.

Each was verified to **fail against the implementation it guards** — the
first four against the original `readBytes`, the EOF-normalization case
against this PR with the conversion removed — so they are regression
tests
rather than tautologies.

They use testify, unlike the rest of `reader_test.go`, which is
`t.Fatalf`
throughout — the other three test files in the package use testify, and
`require.ErrorIs(err, io.ErrUnexpectedEOF)` has no clean `t.Fatalf`
form. Say
the word if you would rather the file stay internally consistent.

- `cd sdk && go test ./... -race` — green
- `golangci-lint run ./internal/zipstream/...` — 0 issues
- `make fmt` — no diff

---------

Signed-off-by: Dave Mihalcik <dmihalcik@virtru.com>

…ap (opentdf#3945)

> **Part 16 of 20** in the DSPX-2604 re-cut. Base branch:
`dspx-2604-07-readfull`.
>
> This stack replaces opentdf#3782 / opentdf#3865 / opentdf#3921, which stay open and
untouched
> until it lands. Nothing here is a rebase of those branches — the work
was
> re-cut from the ticket so each PR stands on its own.

### Proposed Changes

CreateTDF and CreateTDFContext took an io.ReadSeeker, so a caller with a
pipe,
a socket, or any other one-pass source had to spool the whole payload to
disk
or memory first. That is the block DSPX-2604 exists to remove: the
Everfox
re-wrap pipeline hands us a stream it cannot rewind. Both now take an
io.Reader and consume it from its current position through EOF.

Seekability was only ever used to measure the input. The length still
matters,
but it is now resolved rather than required:

- WithInputSize(n) declares it outright, for a reader that cannot report
it;
- failing that, a reader that happens to implement io.Seeker is probed,
and
    the cursor restored to wherever it was;
  - failing both, the payload is unmeasurable and is read until it ends.

The one thing an unmeasurable payload gives up is the compact ZIP32
layout.
The ZIP64 decision is baked into the payload's local file header, which
is
emitted ahead of the first segment, so it cannot be revisited once the
archive
has started; a payload that might exceed a 32-bit offset has to be
written as
ZIP64 from the outset. WithInputSize exists to buy that back — declaring
the
length of a piped payload keeps it in ZIP32 when it fits.

The read loop no longer computes a segment count up front. It reads a
buffer
at a time until EOF, which is what makes an unknown length workable, and
happens to be the same code path for a short final segment. An empty
payload
still produces one empty segment. The segment count is still passed to
the
archive writer when it is known, because that is what keeps a large
declared
count from being clamped to a one-segment capacity hint.

Three behavior changes worth calling out. The first is the only one that
can
silently shorten a payload; the other two fail loudly.

- A seekable reader is no longer rewound. The old code seeked to the end
to
measure and then back to byte 0, so it encrypted the whole file no
matter
where the caller had left the cursor. resolveInputSize saves the current
position and restores that one, so the payload is whatever remains from
    there. Concretely: a caller that sniffs the first 512 bytes for
content-type detection and then hands the same *os.File to CreateTDF
used
to get the whole file and now gets the file minus 512 bytes — no compile
error, no runtime error, just a shorter TDF that looks complete. Reading
from the current position is the right semantics for an io.Reader-shaped
API, and `a seekable reader is encrypted from its current position` pins
    it, but a caller relying on the rewind has to seek to 0 itself now.

- The 64 GB cap (maxFileSizeSupported/errFileTooLarge) is gone. It could
only ever be enforced on a measurable payload, so keeping it would have
meant `encrypt bigfile` failing where `encrypt < bigfile` succeeded.
Both
    were unexported; nothing outside the package referenced them.

- A declared size is exact, not an upper bound. A reader that reaches
EOF
    early now fails the call with errInputShorterThanDeclared instead of
returning a TDF that is silently short of the payload the caller asked
to
encrypt. Reading still stops at the declared size if the reader has
more.

What bounds a payload now, with the cap gone. maxPayloadSegments refuses
a size
needing more than MaxInt32 segments, but segmentCount is the only place
it is
enforced and that runs only when the length resolves — an unmeasurable
stream
has no segment ceiling. Underneath that, the real limit is manifest
memory: one
manifest segment and one archive-writer entry per segment, all live
until
finalize, roughly 500k entries per terabyte at the 2 MiB default.
Neither is
reachable today — 4 PiB at that segment size — so this is a note about
where
the wall moved to, not a regression. DSPX-4905 tracks bounding and
measuring
both.

Testing: Test_CreateTDF_StreamingInput covers the three measurement
modes
across empty, sub-segment, exact-multiple, and partial-final-segment
payloads,
asserting the ZIP64 choice, the segment count, and a full round trip
through
LoadTDF. Test_CreateTDF_InputSizeBounds covers the negative, over-long,
short,
and mid-stream-start cases. Both guards were mutation-checked: removing
the
io.LimitReader fails "declared size bounds the read", and dropping the
unknown-size ZIP64 rule fails every unmeasurable case.

### Checklist

- [x] I have added or updated unit tests
- [ ] I have added or updated integration tests (if appropriate)
- [ ] I have added or updated documentation

### Testing Instructions

```
cd sdk && go test ./... -race
```

`Test_CreateTDF_StreamingInput` and `Test_CreateTDF_InputSizeBounds` are
the
new coverage. Both guards were mutation-checked: removing the
`io.LimitReader`
fails "declared size bounds the read", and dropping the unknown-size
ZIP64 rule
fails every unmeasurable case.

<details>
<summary><b>The full DSPX-2604 stack — 20 PRs</b></summary>

| # | PR | Based on |
|---|----|----------|
| 01 | opentdf#3930 chore: bump go.work toolchain to go1.25.12 and simplify an
rt_test condition | `main` |
| 02 | opentdf#3931 feat(sdk): make the zipstream clock injectable for
deterministic ZIP output | `main` |
| 03 | opentdf#3932 fix(sdk): reject a zipstream write set that omits segment 0
| opentdf#3931 |
| 04 | opentdf#3933 fix(sdk): map ReadAt plaintext offsets from cumulative
segment sizes | `main` |
| 05 | opentdf#3934 chore(sdk): extract integrityAlgorithmString,
createPolicyBinding, signAssertions | `main` |
| 06 | opentdf#3935 chore(sdk): add direct tests for createKeyAccess,
encryptMetadata and tdfSalt | `main` |
| 07 | opentdf#3936 fix(sdk): fill each segment with io.ReadFull and size the
buffer to the input | `main` |
| 08 | opentdf#3937 chore(cli): move streaming IO helpers into pkg | `main` |
| 09 | opentdf#3938 fix(cli): stream encrypt instead of buffering the whole
payload | opentdf#3937 |
| 10 | opentdf#3939 fix(cli): stream decrypt and inspect instead of buffering |
opentdf#3938 |
| 11 | opentdf#3940 feat(sdk): add a chunked segment writer (experimental) |
`dspx-2604-base-11` = opentdf#3932 + opentdf#3934 + opentdf#3935 |
| 12 | opentdf#3941 fix(sdk): stop GetManifest from splitting the key under the
lock | opentdf#3940 |
| 13 | opentdf#3942 fix(sdk): reject a chunked split naming a KAS with no
resolved public key | opentdf#3941 |
| 14 | opentdf#3943 chore(sdk): alias experimental/tdf manifest and assertion
types | opentdf#3942 |
| 15 | opentdf#3944 fix(sdk): emit spec-compliant key access in
experimental/tdf and delegate Writer | opentdf#3943 |
| 16 | opentdf#3945 feat(sdk): accept io.Reader in CreateTDF and drop the 64 GB
payload cap | opentdf#3936 |
| 17 | opentdf#3946 chore(sdk): rewrite CreateTDF on top of the chunked writer
| `dspx-2604-base-17` = opentdf#3944 + opentdf#3945 |
| 18 | opentdf#3947 chore(sdk): drop dead TDFConfig fields and deprecate the
TDFFormat enum | opentdf#3946 |
| 19 | opentdf#3948 fix(cli): drop the encrypt-side stdin spool |
`dspx-2604-base-19` = opentdf#3947 + opentdf#3939 |
| 20 | opentdf#3949 feat(sdk): graduate the chunked writer to stable API |
opentdf#3948 |

**Reviewable in parallel right now**, since they sit directly on `main`
and depend on
nothing else: 01, 02, 04, 05, 06, 07, 08.

**Why three PRs have a `dspx-2604-base-*` base.** A GitHub PR takes one
base branch,
but 11, 17 and 19 each build on more than one parent. The `base-*`
branches are empty
merge commits that exist only to join those parents so the PR diff shows
exactly its
own change and nothing else. They contain no code, have no PR of their
own, and go
away once their parents land — retarget the child onto `main` at that
point.

**Wants a cross-SDK xtest run before merge:** 15, 17 (and therefore 20).
They touch
the KAS wire format.

**Red checks you may see are network flakes, not this stack.** Four
distinct ones hit
this batch and all clear on re-run: `golangci-lint config verify` timing
out on
`https://golangci-lint.run/.../golangci.v2.8.jsonschema.json` (fails the
whole `go
(<module>)` job and fail-fast cancels its siblings), the bats installer
getting a 403,
Docker Hub timing out on `keycloak/keycloak:26.4`, and `buf` reporting
"the server
hosted at that remote is unavailable" while the Java SDK generates
sources. The
`govulncheck` step also emits `##[error]` annotations against the
go1.25.11 stdlib, but
it is `continue-on-error: true` and never fails a job — 01 bumps the
toolchain and
clears those annotations.

</details>


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* TDF creation now supports streaming inputs, including non-seekable
readers such as pipes and network streams.
* Added `WithInputSize` to declare the exact payload size when it cannot
be determined automatically.
* Payloads are processed in segments, supporting unknown sizes and ZIP64
archives when needed.

* **Bug Fixes**
* Invalid or inaccurate declared sizes now result in clear errors,
including when input ends too early.
  * Payloads exceeding the supported segment limit are rejected.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

Signed-off-by: Dave Mihalcik <dmihalcik@virtru.com>

> **Part 17 of 20** in the DSPX-2604 re-cut. Base branch:
`dspx-2604-base-17`.
>
> This stack replaces opentdf#3782 / opentdf#3865 / opentdf#3921, which stay open and
untouched
> until it lands. Nothing here is a rebase of those branches — the work
was
> re-cut from the ticket so each PR stands on its own.

### Proposed Changes

CreateTDFContext and the chunked writer had grown two full
implementations of
the same thing: build a policy, split the DEK, wrap each share to its
KAS,
encrypt segments, accumulate an aggregate hash, sign assertions, emit a
manifest. Two copies of TDF construction is one too many — every spec
change
has to land twice, and the second copy is the one that gets forgotten.

CreateTDF now delegates. It keeps the parts that are genuinely its own —
the
KAO template, autoconfigure, the input-size resolution and read loop
from the
previous commit — and hands each segment to the chunked writer, which
owns
manifest assembly for both paths from here on.

The two paths differ in when key access is resolved, so that is what the
writer is now parameterized on. The chunked writer defers to a
KeySplitter at
Finalize, because a caller may still be adding attributes while segments
are
in flight. CreateTDF cannot: it knows its attributes up front and wants
an
unreachable KAS to fail the call before a single payload byte reaches
the
output writer. Both are expressed as a keyAccessResolver, with the DEK
now
injectable so CreateTDF can wrap it ahead of time and hand the writer a
staticKeyAccess.

Everything downstream of that split is shared: resolvePolicyAndKeyAccess
and
buildKeyAccessObjects replace prepareManifest's inline loop and the
chunked
writer's buildChunkedPolicy/buildChunkedKeyAccessObjects, so both paths
now
emit byte-identical policy and key access for the same attributes.

Two things fall out of the unification, both moving the chunked writer
onto
the shipped classic behavior:

- With zero attributes the policy body's "dataAttributes" and "dissem"
are
    now null rather than []. createPolicyObjectFromFQNs initializes them
    inside the attribute loop; the deleted buildChunkedPolicy did so
    unconditionally. The classic path has always emitted null here.

- A KAS named by a split but missing a public key is still rejected
rather
than skipped (the check moved into buildKeyAccessObjects), but the error
now names the missing PEM rather than the absent map entry — the two
cases
    were indistinguishable in practice and only the outcome matters.

The writer also gained an explicit segment size. It previously reported
the
first segment's actual length as defaultSegmentSize, which is only
correct
when every segment is full; a single-segment TDF would advertise a short
default. CreateTDF knows the configured size and now says so.

TDFObject loses aesGcm and payloadKey, which only ever existed to carry
state
between prepareManifest and the encrypt loop.

Since opentdf#3940 (revised) fixed the chunked writer's root to HS256 and its
segments to GMAC, there is nothing to plumb through here: CreateTDF
stops
passing TDFConfig.rootIntegrityAlg and segmentIntegrityAlg to the
writer.
Those were already the only values the defaults could hold and no
exported
option set either, so the manifest is unchanged.

Deliberately not in this commit: removing enableEncryption, tdfFormat,
readActionName, and the two now-unread integrity fields, which are dead
but
unrelated; they are a separate cleanup (opentdf#3947).

Testing: the existing TDFSuite round trips pin byte-level output across
segment sizes, target modes, and multi-KAS splits, and pass unchanged.
Also
verified against the streaming-input and input-size
coverage added in the previous commit, the experimental chunked writer
suite,
and cross-module builds of examples, otdfctl, service, and tests-bdd.

### Checklist

- [x] I have added or updated unit tests
- [ ] I have added or updated integration tests (if appropriate)
- [ ] I have added or updated documentation

### Testing Instructions

```
cd sdk && go test ./... -race
```

The existing `TDFSuite` round trips pin byte-level output across segment
sizes, target modes and multi-KAS splits, and pass unchanged — that is
the
main assurance here.

This touches key access construction on the shipped path, so it wants a
cross-SDK run before merge. This is the branch to pin it to for the
whole
writer-delegation half of the stack: xtest drives the Go side through
`otdfctl` → `SDK.CreateTDF`, and this is the first commit where that
call
reaches the chunked writer at all.

```
gh workflow run xtest.yml --repo opentdf/tests --ref main \
  -f platform-ref=dspx-2604-17-createtdf-delegates \
  -f otdfctl-ref=dspx-2604-17-createtdf-delegates \
  -f java-ref=main -f js-ref=main
```

`otdfctl-ref` must name the branch, not `main` — otherwise the CLI is
built
against main's `sdk/` and the run passes without executing any of this.
The
job label should read `go@dspx-2604-17-createtdf-delegates`.

<details>
<summary><b>The full DSPX-2604 stack — 20 PRs</b></summary>

| # | PR | Based on |
|---|----|----------|
| 01 | opentdf#3930 chore: bump go.work toolchain to go1.25.12 and simplify an
rt_test condition | `main` |
| 02 | opentdf#3931 feat(sdk): make the zipstream clock injectable for
deterministic ZIP output | `main` |
| 03 | opentdf#3932 fix(sdk): reject a zipstream write set that omits segment 0
| opentdf#3931 |
| 04 | opentdf#3933 fix(sdk): map ReadAt plaintext offsets from cumulative
segment sizes | `main` |
| 05 | opentdf#3934 chore(sdk): extract integrityAlgorithmString,
createPolicyBinding, signAssertions | `main` |
| 06 | opentdf#3935 chore(sdk): add direct tests for createKeyAccess,
encryptMetadata and tdfSalt | `main` |
| 07 | opentdf#3936 fix(sdk): fill each segment with io.ReadFull and size the
buffer to the input | `main` |
| 08 | opentdf#3937 chore(cli): move streaming IO helpers into pkg | `main` |
| 09 | opentdf#3938 fix(cli): stream encrypt instead of buffering the whole
payload | opentdf#3937 |
| 10 | opentdf#3939 fix(cli): stream decrypt and inspect instead of buffering |
opentdf#3938 |
| 11 | opentdf#3940 feat(sdk): add a chunked segment writer (experimental) |
`dspx-2604-base-11` = opentdf#3932 + opentdf#3934 + opentdf#3935 |
| 12 | opentdf#3941 fix(sdk): stop GetManifest from splitting the key under the
lock | opentdf#3940 |
| 13 | opentdf#3942 fix(sdk): reject a chunked split naming a KAS with no
resolved public key | opentdf#3941 |
| 14 | opentdf#3943 chore(sdk): alias experimental/tdf manifest and assertion
types | opentdf#3942 |
| 15 | opentdf#3944 fix(sdk): emit spec-compliant key access in
experimental/tdf and delegate Writer | opentdf#3943 |
| 16 | opentdf#3945 feat(sdk): accept io.Reader in CreateTDF and drop the 64 GB
payload cap | opentdf#3936 |
| 17 | opentdf#3946 chore(sdk): rewrite CreateTDF on top of the chunked writer
| `dspx-2604-base-17` = opentdf#3944 + opentdf#3945 |
| 18 | opentdf#3947 chore(sdk): drop dead TDFConfig fields and deprecate the
TDFFormat enum | opentdf#3946 |
| 19 | opentdf#3948 fix(cli): drop the encrypt-side stdin spool |
`dspx-2604-base-19` = opentdf#3947 + opentdf#3939 |
| 20 | opentdf#3949 feat(sdk): graduate the chunked writer to stable API |
opentdf#3948 |

**Reviewable in parallel right now**, since they sit directly on `main`
and depend on
nothing else: 01, 02, 04, 05, 06, 07, 08.

**Why three PRs have a `dspx-2604-base-*` base.** A GitHub PR takes one
base branch,
but 11, 17 and 19 each build on more than one parent. The `base-*`
branches are empty
merge commits that exist only to join those parents so the PR diff shows
exactly its
own change and nothing else. They contain no code, have no PR of their
own, and go
away once their parents land — retarget the child onto `main` at that
point.

**Wants a cross-SDK xtest run before merge:** 15, 17 (and therefore 20).
They touch
the KAS wire format. Dispatch it against 17 or 20, never 15 on its own:
xtest drives
the Go side through `otdfctl` -> `SDK.CreateTDF`, and 17 is the first
commit where
that call reaches the rewritten writer. Set `otdfctl-ref` to the same
branch as
`platform-ref` -- it defaults to `main`, which builds the CLI against
main's `sdk/`
and makes the run vacuous.

**Red checks you may see are network flakes, not this stack.** Four
distinct ones hit
this batch and all clear on re-run: `golangci-lint config verify` timing
out on
`https://golangci-lint.run/.../golangci.v2.8.jsonschema.json` (fails the
whole `go
(<module>)` job and fail-fast cancels its siblings), the bats installer
getting a 403,
Docker Hub timing out on `keycloak/keycloak:26.4`, and `buf` reporting
"the server
hosted at that remote is unavailable" while the Java SDK generates
sources. The
`govulncheck` step also emits `##[error]` annotations against the
go1.25.11 stdlib, but
it is `continue-on-error: true` and never fails a job — 01 bumps the
toolchain and
clears those annotations.

</details>



<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Refactor**
* TDF creation now uses a consistent process for resolving key access
and encrypting payload segments.
* Key shares are validated and checked for successful reconstruction
before payload data is written.
* Existing encrypted output behavior is preserved, including segment
sizing based on the configured size or the first segment when no size is
configured.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->

…gate Writer (opentdf#3944)

> **Part 15 of 20** in the DSPX-2604 re-cut. Base branch:
`dspx-2604-14-type-aliases`.
>
> This stack replaces opentdf#3782 / opentdf#3865 / opentdf#3921, which stay open and
untouched
> until it lands. Nothing here is a rebase of those branches — the work
was
> re-cut from the ticket so each PR stands on its own.

### Proposed Changes

Deletes `experimental/tdf` TDF building logic, in favor of existing,
more compliant
logic found in root `sdk` package.

The experimental writer built its own key access objects, and for EC KAS
keys it built them wrong in three ways at once. It set keyType
`"eccWrapped"`, but `service/kas/access/rewrap.go` dispatches on the
exact
string `"ec-wrapped"` and has no case for the other spelling. It derived
the wrapping key with HKDF and then XORed the DEK, where the spec and
every
KAS expect AES-GCM under that derived key. And it omitted
`schemaVersion`
from the KAO entirely. Any TDF this package produced against an EC KAS
was
undecryptable, and nothing in the repo caught it because the package
tested
its own output against its own expectations.

The fix is not a patch to that code but its deletion. `key_access.go`
(-266)
goes away and `Writer` delegates to `sdk.NewChunkedWriter`, so key
access
objects come from `sdk.createKeyAccess` -- the same code path
`SDK.CreateTDF` has always used and that the cross-SDK tests exercise.
RSA,
EC, ML-KEM and hybrid wrapping now have exactly one implementation.
`key_access_test.go` (-652) goes with it; equivalent coverage against
the
sdk functions landed earlier in this stack, so nothing is lost.

`writer.go` drops from 680 lines to ~292: `Writer` becomes its config
plus
an inner `sdk.ChunkedWriter` and a `finalized` flag. Manifest assembly,
segment encryption, integrity hashing and assertion signing all move to
the
one implementation. `manifest.go` sheds the `calculateSignature` copy
and
the three constants that only its callers needed.

`keysplit_adapter.go` (+60) is why this is a delegation rather than a
rename. `sdk.DefaultKeySplitter` is single-KAS and ignores attributes;
`keysplit.XORSplitter` evaluates the full ABAC boolean expression and
XOR-splits the DEK across every KAS the resulting clauses require. The
two
result shapes are field-identical, so the adapter is a straight copy.
The
one structural mismatch is where the default KAS enters -- sdk passes it
per
`Split` call, keysplit takes it at construction -- so the splitter is
built
inside `Split`.

API changes callers will notice

`Finalize` error values are aliases of their sdk
counterparts rather than copies, so `errors.Is` matches under either
name.

`WithSegments` no longer requires a contiguous prefix starting at 0.
Indices may be sparse -- a caller mapping fixed index blocks onto S3
multipart uploads writes gaps by construction -- but must still name
written
segments in ascending order and may only drop from the end, because that
is
the order the payload is laid out in.

`WithExcludeVersionFromManifest` is deprecated. It was always a no-op:
the
manifest builder never read the flag. Omitting `schemaVersion` is how a
reader is told the TDF predates 4.3.0, and such a reader then expects
hex-then-base64 signatures, which are decided per segment at write time,
long before Finalize sees the option. `WithTargetMode` sets both
together
and is the replacement.

`WithIntegrityAlgorithm` and `WithSegmentIntegrityAlgorithm` no longer
take
effect, and asking for anything but the default is now an error from
`NewWriter` rather than a silent substitution. The other variants were
unsupported
and resulted in incompatible TDFs/

### Checklist

- [x] I have added or updated unit tests
- [ ] I have added or updated integration tests (if appropriate)
- [x] I have added or updated documentation

### Testing Instructions

```
cd sdk && go test ./... -race
cd examples && go build ./...
```

This changes the KAS wire format for EC keys, so it wants a cross-SDK
run
before merge. Pin it to opentdf#3946, not to this branch — that is the first
point
where `CreateTDF` goes through the changed code, and `otdfctl` is the
only
Go consumer xtest drives:

```
gh workflow run xtest.yml --repo opentdf/tests --ref main \
  -f platform-ref=dspx-2604-17-createtdf-delegates \
  -f otdfctl-ref=dspx-2604-17-createtdf-delegates \
  -f java-ref=main -f js-ref=main
```

`otdfctl-ref` must name the branch too: it defaults to `main`, which
builds
the CLI against main's `sdk/` and makes the run vacuous. Check the job
label
reads `go@<branch>` rather than `go@main`.

<details>
<summary><b>The full DSPX-2604 stack — 20 PRs</b></summary>

| # | PR | Based on |
|---|----|----------|
| 01 | opentdf#3930 chore: bump go.work toolchain to go1.25.12 and simplify an
rt_test condition | `main` |
| 02 | opentdf#3931 feat(sdk): make the zipstream clock injectable for
deterministic ZIP output | `main` |
| 03 | opentdf#3932 fix(sdk): reject a zipstream write set that omits segment 0
| opentdf#3931 |
| 04 | opentdf#3933 fix(sdk): map ReadAt plaintext offsets from cumulative
segment sizes | `main` |
| 05 | opentdf#3934 chore(sdk): extract integrityAlgorithmString,
createPolicyBinding, signAssertions | `main` |
| 06 | opentdf#3935 chore(sdk): add direct tests for createKeyAccess,
encryptMetadata and tdfSalt | `main` |
| 07 | opentdf#3936 fix(sdk): fill each segment with io.ReadFull and size the
buffer to the input | `main` |
| 08 | opentdf#3937 chore(cli): move streaming IO helpers into pkg | `main` |
| 09 | opentdf#3938 fix(cli): stream encrypt instead of buffering the whole
payload | opentdf#3937 |
| 10 | opentdf#3939 fix(cli): stream decrypt and inspect instead of buffering |
opentdf#3938 |
| 11 | opentdf#3940 feat(sdk): add a chunked segment writer (experimental) |
`dspx-2604-base-11` = opentdf#3932 + opentdf#3934 + opentdf#3935 |
| 12 | opentdf#3941 fix(sdk): stop GetManifest from splitting the key under the
lock | opentdf#3940 |
| 13 | opentdf#3942 fix(sdk): reject a chunked split naming a KAS with no
resolved public key | opentdf#3941 |
| 14 | opentdf#3943 chore(sdk): alias experimental/tdf manifest and assertion
types | opentdf#3942 |
| 15 | opentdf#3944 fix(sdk): emit spec-compliant key access in
experimental/tdf and delegate Writer | opentdf#3943 |
| 16 | opentdf#3945 feat(sdk): accept io.Reader in CreateTDF and drop the 64 GB
payload cap | opentdf#3936 |
| 17 | opentdf#3946 chore(sdk): rewrite CreateTDF on top of the chunked writer
| `dspx-2604-base-17` = opentdf#3944 + opentdf#3945 |
| 18 | opentdf#3947 chore(sdk): drop dead TDFConfig fields and deprecate the
TDFFormat enum | opentdf#3946 |
| 19 | opentdf#3948 fix(cli): drop the encrypt-side stdin spool |
`dspx-2604-base-19` = opentdf#3947 + opentdf#3939 |
| 20 | opentdf#3949 feat(sdk): graduate the chunked writer to stable API |
opentdf#3948 |

**Reviewable in parallel right now**, since they sit directly on `main`
and depend on
nothing else: 01, 02, 04, 05, 06, 07, 08.

**Why three PRs have a `dspx-2604-base-*` base.** A GitHub PR takes one
base branch,
but 11, 17 and 19 each build on more than one parent. The `base-*`
branches are empty
merge commits that exist only to join those parents so the PR diff shows
exactly its
own change and nothing else. They contain no code, have no PR of their
own, and go
away once their parents land — retarget the child onto `main` at that
point.

**Wants a cross-SDK xtest run before merge:** 15, 17 (and therefore 20).
They touch
the KAS wire format. Dispatch it against 17 or 20, never 15 on its own:
xtest drives
the Go side through `otdfctl` -> `SDK.CreateTDF`, and 17 is the first
commit where
that call reaches the rewritten writer. Set `otdfctl-ref` to the same
branch as
`platform-ref` -- it defaults to `main`, which builds the CLI against
main's `sdk/`
and makes the run vacuous.

**Red checks you may see are network flakes, not this stack.** Four
distinct ones hit
this batch and all clear on re-run: `golangci-lint config verify` timing
out on
`https://golangci-lint.run/.../golangci.v2.8.jsonschema.json` (fails the
whole `go
(<module>)` job and fail-fast cancels its siblings), the bats installer
getting a 403,
Docker Hub timing out on `keycloak/keycloak:26.4`, and `buf` reporting
"the server
hosted at that remote is unavailable" while the Java SDK generates
sources. The
`govulncheck` step also emits `##[error]` annotations against the
go1.25.11 stdlib, but
it is `continue-on-error: true` and never fails a job — 01 bumps the
toolchain and
clears those annotations.

</details>



<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
  * Added an option to target specific TDF specification versions.
* Added support for XOR-based key splitting in the experimental TDF
writer.
* **Bug Fixes**
* Unsupported integrity algorithms are now rejected when creating a
writer; GMAC is the only supported segment integrity algorithm.
  * Improved compatibility of writer errors with the stable SDK.
* **Documentation**
* Updated guidance for sparse segments, finalized output, integrity
algorithms, architecture, and thread safety.
* **Deprecations**
  * `WithExcludeVersionFromManifest` is deprecated and has no effect.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Signed-off-by: Dave Mihalcik <dmihalcik@virtru.com>

…um (opentdf#3947)

> **Part 18 of 20** in the DSPX-2604 re-cut. Base branch:
`dspx-2604-17-createtdf-delegates`.
>
> This stack replaces opentdf#3782 / opentdf#3865 / opentdf#3921, which stay open and
untouched
> until it lands. Nothing here is a rebase of those branches — the work
was
> re-cut from the ticket so each PR stands on its own.

### Proposed Changes

TDFConfig.enableEncryption was set to true at construction and never
read
again; nothing could turn it off and nothing branched on it. tdfFormat
was
likewise fixed at JSONFormat forever. readActionName was a leftover
constant
with no references. All three are gone.

rootIntegrityAlg and segmentIntegrityAlg go with them. opentdf#3940 (revised)
fixed
the chunked writer's root to HS256 and its segments to GMAC, and opentdf#3946
made
CreateTDF delegate to it, so nothing reads either field any more. No
exported
option ever set one, and the defaults they held were exactly those two
algorithms, so the manifest is byte-identical. They are unexported, so
they
are deleted outright rather than deprecated.

`TestIntegrityAlgDefaults` asserted on those fields and could not
survive
them. The invariant it guarded is now asserted on a manifest `CreateTDF`
actually produced — `TDFSuite.testEncrypt` checks `manifest.Algorithm`
is
HS256 and `manifest.SegmentHashAlgorithm` is GMAC — so every encrypt
case in
the suite carries it rather than one test reading back a struct field.

TDFFormat, JSONFormat, and XMLFormat are exported, so they are
deprecated
rather than deleted. XML manifests were never implemented and the enum
has no
remaining consumer inside the SDK.

### Checklist

- [x] I have added or updated unit tests
- [ ] I have added or updated integration tests (if appropriate)
- [x] I have added or updated documentation

### Testing Instructions

```
make build && make test
```

Nothing reads any of the removed fields; the check is that the tree
still
builds across `sdk`, `service`, `otdfctl`, `examples` and `tests-bdd`,
and
that `TDFSuite` still emits HS256/GMAC on every encrypt case.

<details>
<summary><b>The full DSPX-2604 stack — 20 PRs</b></summary>

| # | PR | Based on |
|---|----|----------|
| 01 | opentdf#3930 chore: bump go.work toolchain to go1.25.12 and simplify an
rt_test condition | `main` |
| 02 | opentdf#3931 feat(sdk): make the zipstream clock injectable for
deterministic ZIP output | `main` |
| 03 | opentdf#3932 fix(sdk): reject a zipstream write set that omits segment 0
| opentdf#3931 |
| 04 | opentdf#3933 fix(sdk): map ReadAt plaintext offsets from cumulative
segment sizes | `main` |
| 05 | opentdf#3934 chore(sdk): extract integrityAlgorithmString,
createPolicyBinding, signAssertions | `main` |
| 06 | opentdf#3935 chore(sdk): add direct tests for createKeyAccess,
encryptMetadata and tdfSalt | `main` |
| 07 | opentdf#3936 fix(sdk): fill each segment with io.ReadFull and size the
buffer to the input | `main` |
| 08 | opentdf#3937 chore(cli): move streaming IO helpers into pkg | `main` |
| 09 | opentdf#3938 fix(cli): stream encrypt instead of buffering the whole
payload | opentdf#3937 |
| 10 | opentdf#3939 fix(cli): stream decrypt and inspect instead of buffering |
opentdf#3938 |
| 11 | opentdf#3940 feat(sdk): add a chunked segment writer (experimental) |
`dspx-2604-base-11` = opentdf#3932 + opentdf#3934 + opentdf#3935 |
| 12 | opentdf#3941 fix(sdk): stop GetManifest from splitting the key under the
lock | opentdf#3940 |
| 13 | opentdf#3942 fix(sdk): reject a chunked split naming a KAS with no
resolved public key | opentdf#3941 |
| 14 | opentdf#3943 chore(sdk): alias experimental/tdf manifest and assertion
types | opentdf#3942 |
| 15 | opentdf#3944 fix(sdk): emit spec-compliant key access in
experimental/tdf and delegate Writer | opentdf#3943 |
| 16 | opentdf#3945 feat(sdk): accept io.Reader in CreateTDF and drop the 64 GB
payload cap | opentdf#3936 |
| 17 | opentdf#3946 chore(sdk): rewrite CreateTDF on top of the chunked writer
| `dspx-2604-base-17` = opentdf#3944 + opentdf#3945 |
| 18 | opentdf#3947 chore(sdk): drop dead TDFConfig fields and deprecate the
TDFFormat enum | opentdf#3946 |
| 19 | opentdf#3948 fix(cli): drop the encrypt-side stdin spool |
`dspx-2604-base-19` = opentdf#3947 + opentdf#3939 |
| 20 | opentdf#3949 feat(sdk): graduate the chunked writer to stable API |
opentdf#3948 |

**Reviewable in parallel right now**, since they sit directly on `main`
and depend on
nothing else: 01, 02, 04, 05, 06, 07, 08.

**Why three PRs have a `dspx-2604-base-*` base.** A GitHub PR takes one
base branch,
but 11, 17 and 19 each build on more than one parent. The `base-*`
branches are empty
merge commits that exist only to join those parents so the PR diff shows
exactly its
own change and nothing else. They contain no code, have no PR of their
own, and go
away once their parents land — retarget the child onto `main` at that
point.

**Wants a cross-SDK xtest run before merge:** 15, 17 (and therefore 20).
They touch
the KAS wire format.

**Red checks you may see are network flakes, not this stack.** Four
distinct ones hit
this batch and all clear on re-run: `golangci-lint config verify` timing
out on
`https://golangci-lint.run/.../golangci.v2.8.jsonschema.json` (fails the
whole `go
(<module>)` job and fail-fast cancels its siblings), the bats installer
getting a 403,
Docker Hub timing out on `keycloak/keycloak:26.4`, and `buf` reporting
"the server
hosted at that remote is unavailable" while the Java SDK generates
sources. The
`govulncheck` step also emits `##[error]` annotations against the
go1.25.11 stdlib, but
it is `continue-on-error: true` and never fails a job — 01 bumps the
toolchain and
clears those annotations.

</details>



<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Bug Fixes**
* Policy creation now returns a validation error when supplied with an
empty fully qualified name, rather than proceeding with invalid input.

* **Configuration**
* TDF configuration no longer exposes settings for enabling encryption,
choosing a format, or selecting root and segment integrity algorithms.
The format constants are marked deprecated; XML format is documented as
never implemented. Existing applications that set these options may need
to update their configuration.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

> **Part 19 of 20** in the DSPX-2604 re-cut. Base branch:
`dspx-2604-base-19`.
>
> This stack replaces opentdf#3782 / opentdf#3865 / opentdf#3921, which stay open and
untouched
> until it lands. Nothing here is a rebase of those branches — the work
was
> re-cut from the ticket so each PR stands on its own.

### Proposed Changes

DSPX-4499 fixed encrypt's OOM by streaming, but CreateTDF still required
an
io.ReadSeeker, so piped stdin had to be spooled to a temporary file
first. That
traded the whole-payload allocation for a disk write and a writable
TMPDIR --
better, but not the point. Now that CreateTDF takes an io.Reader, the
pipe goes
straight to the SDK.

A file is still opened seekably, on purpose. The SDK measures a seekable
payload and keeps the archive in the compact ZIP32 layout; an
unmeasurable one
has to be ZIP64, because the choice is baked into the payload's local
file
header before the first segment goes out. So `encrypt file.txt` is
unchanged
byte for byte, and `... | encrypt` produces a slightly larger TDF than
it did
when it was spooled. That is the trade, and it is the right way round:
nobody
should need a writable temp directory to encrypt a stream.

MIME sniffing is what made this more than a deletion. It reads the first
megabyte and previously seeked back to zero, which a pipe cannot do.
Wrapping
the input in a bufio.Reader is not an option either -- that hides the
Seeker
and would silently flip every file encrypt to ZIP64. detectMimeType now
returns
a reader alongside the type: the same reader for a seekable input,
rewound; the
sniffed prefix pushed back with io.MultiReader for anything else. A
megabyte in
memory at most, and only when --mime-type was not given.

Testing: TestDetectMimeTypePreservesThePayload now runs each case twice,
once
over a reader whose Seek method is hidden, and asserts the payload
arrives
whole either way. TestDetectMimeTypeKeepsSeekability guards the ZIP32
layout
directly, since nothing else would fail if a file came back wrapped. e2e
gains
"encrypt measures a file and streams a pipe", which reads the local file
header's extra field length to tell the two layouts apart -- both forms
round-trip, so a quiet return to spooling would show up nowhere else.

### Checklist

- [x] I have added or updated unit tests
- [x] I have added or updated integration tests (if appropriate)
- [x] I have added or updated documentation

### Testing Instructions

```
cd otdfctl && go test ./... -race
```

e2e, against a running platform:

```
cd otdfctl && bats e2e/streaming.bats
```

The file is tagged `payload_streaming` and runs in its own pass ahead of
the
parallel batch — see opentdf#3939 for why that ordering is load-bearing (the
platform
base key `key-base.bats` sets cannot be un-set, and breaks every
unattributed
encrypt scheduled after it).

The case to look at is `encrypt measures a file and streams a pipe`,
which
asserts the ZIP layout rather than the round trip: a file must stay
ZIP32
(extra field length 0 at offset 28 of the payload's local file header)
and a
pipe must be ZIP64 (non-zero). A quiet return to spooling would show up
nowhere
else, since both forms round-trip fine.

<details>
<summary><b>The full DSPX-2604 stack — 20 PRs</b></summary>

| # | PR | Based on |
|---|----|----------|
| 01 | opentdf#3930 chore: bump go.work toolchain to go1.25.12 and simplify an
rt_test condition | `main` |
| 02 | opentdf#3931 feat(sdk): make the zipstream clock injectable for
deterministic ZIP output | `main` |
| 03 | opentdf#3932 fix(sdk): reject a zipstream write set that omits segment 0
| opentdf#3931 |
| 04 | opentdf#3933 fix(sdk): map ReadAt plaintext offsets from cumulative
segment sizes | `main` |
| 05 | opentdf#3934 chore(sdk): extract integrityAlgorithmString,
createPolicyBinding, signAssertions | `main` |
| 06 | opentdf#3935 chore(sdk): add direct tests for createKeyAccess,
encryptMetadata and tdfSalt | `main` |
| 07 | opentdf#3936 fix(sdk): fill each segment with io.ReadFull and size the
buffer to the input | `main` |
| 08 | opentdf#3937 chore(cli): move streaming IO helpers into pkg | `main` |
| 09 | opentdf#3938 fix(cli): stream encrypt instead of buffering the whole
payload | opentdf#3937 |
| 10 | opentdf#3939 fix(cli): stream decrypt and inspect instead of buffering |
opentdf#3938 |
| 11 | opentdf#3940 feat(sdk): add a chunked segment writer (experimental) |
`dspx-2604-base-11` = opentdf#3932 + opentdf#3934 + opentdf#3935 |
| 12 | opentdf#3941 fix(sdk): stop GetManifest from splitting the key under the
lock | opentdf#3940 |
| 13 | opentdf#3942 fix(sdk): reject a chunked split naming a KAS with no
resolved public key | opentdf#3941 |
| 14 | opentdf#3943 chore(sdk): alias experimental/tdf manifest and assertion
types | opentdf#3942 |
| 15 | opentdf#3944 fix(sdk): emit spec-compliant key access in
experimental/tdf and delegate Writer | opentdf#3943 |
| 16 | opentdf#3945 feat(sdk): accept io.Reader in CreateTDF and drop the 64 GB
payload cap | opentdf#3936 |
| 17 | opentdf#3946 chore(sdk): rewrite CreateTDF on top of the chunked writer
| `dspx-2604-base-17` = opentdf#3944 + opentdf#3945 |
| 18 | opentdf#3947 chore(sdk): drop dead TDFConfig fields and deprecate the
TDFFormat enum | opentdf#3946 |
| 19 | opentdf#3948 fix(cli): drop the encrypt-side stdin spool |
`dspx-2604-base-19` = opentdf#3947 + opentdf#3939 |
| 20 | opentdf#3949 feat(sdk): graduate the chunked writer to stable API |
opentdf#3948 |

**Reviewable in parallel right now**, since they sit directly on `main`
and depend on
nothing else: 01, 02, 04, 05, 06, 07, 08.

**Why three PRs have a `dspx-2604-base-*` base.** A GitHub PR takes one
base branch,
but 11, 17 and 19 each build on more than one parent. The `base-*`
branches are empty
merge commits that exist only to join those parents so the PR diff shows
exactly its
own change and nothing else. They contain no code, have no PR of their
own, and go
away once their parents land — retarget the child onto `main` at that
point.

**Wants a cross-SDK xtest run before merge:** 15, 17 (and therefore 20).
They touch
the KAS wire format.

**Red checks you may see are network flakes, not this stack.** Four
distinct ones hit
this batch and all clear on re-run: `golangci-lint config verify` timing
out on
`https://golangci-lint.run/.../golangci.v2.8.jsonschema.json` (fails the
whole `go
(<module>)` job and fail-fast cancels its siblings), the bats installer
getting a 403,
Docker Hub timing out on `keycloak/keycloak:26.4`, and `buf` reporting
"the server
hosted at that remote is unavailable" while the Java SDK generates
sources. The
`govulncheck` step also emits `##[error]` annotations against the
go1.25.11 stdlib, but
it is `continue-on-error: true` and never fails a job — 01 bumps the
toolchain and
clears those annotations.

</details>


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Encryption now supports piped and other non-seekable input, including
larger streams.
* Inputs that cannot be measured in advance produce ZIP64 output;
seekable files continue to use ZIP32 when applicable.
* Regular-file redirects can be read directly without being copied to a
temporary file.
  * Decryption now supports TDFs supplied through pipes and redirects.
* Encrypted output remains compatible with the existing decryption flow.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

Signed-off-by: Dave Mihalcik <dmihalcik@virtru.com>

🤖 I have created a release *beep* *boop*
---


##
[0.33.0](opentdf/platform@sdk/v0.32.0...sdk/v0.33.0)
(2026-09-25)


### Features

* **sdk:** accept io.Reader in CreateTDF and drop the 64 GB payload cap
([opentdf#3945](opentdf#3945))
([20ff505](opentdf@20ff505))
* **sdk:** add a chunked segment writer (experimental)
([opentdf#3940](opentdf#3940))
([69cb391](opentdf@69cb391))


### Bug Fixes

* **deps:** bump google.golang.org/grpc from 1.83.1 to 1.83.2 in /sdk
([opentdf#4018](opentdf#4018))
([d1d4bd6](opentdf@d1d4bd6))
* **sdk:** emit spec-compliant key access in experimental/tdf and
delegate Writer
([opentdf#3944](opentdf#3944))
([3e8d88a](opentdf@3e8d88a))
* **sdk:** read zip entry bytes with io.ReadFull (DSPX-4590)
([opentdf#4087](opentdf#4087))
([40fa788](opentdf@40fa788))
* **sdk:** stop GetManifest from splitting the key under the lock
([opentdf#3941](opentdf#3941))
([cf1b083](opentdf@cf1b083))

---
This PR was generated with [Release
Please](https://github.com/googleapis/release-please). See
[documentation](https://github.com/googleapis/release-please#release-please).

Co-authored-by: opentdf-automation[bot] <149537512+opentdf-automation[bot]@users.noreply.github.com>